Autonomous Wide-Angle License Plate Recognition

ABSTRACT

A system in a moving surveillance vehicle operates in background mode to capture images of license plates of neighboring moving vehicles, which may occupy lanes other than the lane in which the surveillance vehicle is moving. The images are used to determine the license plate numbers of the moving vehicles, which are then checked against a database to determine whether there are any potential law enforcement-related problems that require the attention of the operator. If so, the system alerts the operator using an audible tone, visual prompt, vibration, or in some other suitable manner. The entire process, including generation of the alert can occur autonomously of the operator.

This application is a continuation of U.S. patent application Ser. No.10/546,555 filed Aug. 22, 2005 which is a U.S. National Phase ofPCT/US03/21958 filed Jul. 10, 2003. These and all other extrinsicmaterials discussed herein are incorporated by reference in theirentirety. Where a definition or use of a term in an incorporatedreference is inconsistent or contrary to the definition of that termprovided herein, the definition of that term provided herein applies andthe definition of that term in the reference does not apply.

FIELD OF THE INVENTION

The present invention relates generally to license plate recognition.

BACKGROUND OF THE INVENTION

Traffic police, highway patrol and other mobile security personnel havea need to accurately and efficiently identify potential law enforcementproblems with respect to nearby motor vehicles. One well-recognizedstrategy is to “run” license plate numbers of such vehicles against adatabase.

It is a well known practice for an officer in a patrol car to visuallyread a license plate of a target vehicle, and then call in the number toa support center. It is also known for an officer to utilize an on-boarddigital video camera to capture an image of the vehicle license plate ofa parked vehicle, and even to use an on-board computer to analyze theimage to determine the license plate number. In both cases, however,conscious effort is required on the part of the officer, which divertsattention from driving or other activities. In some cases the diversionof attention can be dangerous, and can even contribute to an otherwisepreventable car accident.

A highly advanced license plate recognition and checking system isdescribed in Japanese patent 11-296785, published on Oct. 29, 1999, thedisclosure of which is incorporated herein in its entirety by reference.In that system the patrol car is provided with an on-board camera andcamera controller, optical character recognition (OCR) software, adatabase, and a display device. While moving along a road, the driverplaces the patrol car in front of or behind the target car, and thentriggers operation of the camera. The camera photographs the licenseplate of a preceding or following target vehicle, and sends the capturedimage to the on-board computer. OCR software running on the computerdetermines the license plate number, and applies that number against theon-board database. The results are displayed to the operator.

Apparently, the inventors of Japanese patent 11-296785 did notappreciate that (a) forcing the driver of the patrol car to maneuver hisvehicle relative to the target vehicle, and (b) forcing the operator totrigger the camera, involve conscious efforts on the part of the driver,which preclude the system from operating in a truly autonomous fashion.The issue of being fully autonomous is not a mere design choice. Fullyautonomous operation is not only safer and more thorough, it also has aparticularly important function in countering accusations of racialprofiling. If a system could be devised that would autonomously processlicense plate information of all vehicles in view of the camera,including vehicles in other lanes, then the operator could not bereasonably accused of focusing on any particular car or driver.

Thus, there is still a need for more fully autonomous surveillance ofmoving automotive vehicles from another moving vehicle.

SUMMARY OF THE INVENTION

The present invention is generally directed to systems and methods inwhich a first moving vehicle (surveillance vehicle) captures vehiclelicense plate number information from a second moving vehicle in a fullyautonomous fashion, and applies that information against a database toidentify potential problems with the second vehicle or its driver. Inpreferred embodiments the surveillance vehicle is preferably a policecar or other patrol unit; and the second vehicles can be in any lanethat is visible from the camera, including the same lane as thatoccupied by the first vehicle, other driving lanes, and even in parkingareas or road shoulders.

A digital video camera with a wide-angle lens is preferably used tocapture the license place information. Alternatively, any other suitableimage-capturing device could be used, including a still-image camera.The camera can advantageously be mounted on the front, rear, side or topof the surveillance vehicle, and preferably has a viewing angle range ofat least 120°.

An on-board processor can either (a) perform optical characterrecognition on an acquired image to determine the license plate number,or (b) transmit at least a portion of the image to a remote site forthat determination. The license plate number can then be applied againstone or more databases, which can be local and/or remote to the cameraand processor, to identify potential problems with the car or driver ofthe second vehicle.

Ideally, each of these functions is carried out continuously, inbackground mode without any input from the driver or other operator ofthe vehicle carrying the system. When potential problems are identified,the operator is notified using an auditory, visual or other alert.

Various objects, features, aspects and advantages of the presentinvention will become more apparent from the following detaileddescription of preferred embodiments of the invention, along with theaccompanying drawings in which like numerals represent like components.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a side perspective view of a moving surveillance vehiclecarrying a surveillance system according to the inventive subjectmatter, with neighboring vehicles in various lanes.

FIG. 2 is a schematic representation of the surveillance system of FIG.1.

DETAILED DESCRIPTION

FIG. 1 generally depicts a surveillance vehicle 10 equipped with asurveillance system 20 (see FIG. 2) that is driving along a roadway 30in lane 32. Another vehicle 40 is traveling ahead of surveillancevehicle 10 in lane 22, while vehicles 42, 44 are traveling in the samedirection as surveillance vehicle 10 in lane 34. Vehicles 46 and 48 aretraveling in the oncoming direction in lane 36, across low median 35.Surveillance system 20 generally comprises first and second digitalvideo cameras 22A, 22B, a processor 24 (see FIG. 2), and a signalgenerator 28 (see FIG. 2).

The surveillance vehicle 10 will often be a patrol car 10, but can beany road or off-road vehicle, including jeeps, trucks, ambulances,buses, recreational vehicles, fire engines, and so forth. Theneighboring vehicles 40, 42, 44, 46 and 48 can likewise be anycombination of any types of vehicles, and will obviously be dispersedaround the patrol car 10 in a manner that varies infinitely over time.Most of the neighboring vehicles 40, 42, 44, 46 and 48 will have rearlicense plates, and some can have front license plates. In this drawing,the relevant license plates for vehicles 40, 42, 44, 46 and 48 are 40A,42A, 44A, 46A, and 48A, respectively.

Cameras 22A, 22B are mounted at the front and rear portions ofsurveillance vehicle 10. Mounting can be on the bumpers or anywhereelse, and can even be located in other positions such as in the sirentower on top of the surveillance vehicle 10 or inside the cab behind thewindshield. One or both of cameras 22A, 22B can be mounted in the centerline of the surveillance vehicle 10, or off-center in any suitablemanner. There must of course be at least one camera, which could providefront, rear, side or combination coverage. Second, third, and othercameras are optional. A person skilled in the art should recognize thatmore than two cameras could be mounted on surveillance vehicle 10 insuitable locations (e.g., front, rear, side or top of vehicle) to allowup to 360° license plate scan coverage.

Each camera 22A, 22B has a lens (not shown) that “sees” license plateswithin their respective viewing angles 24A, 24B. A wide-angle lens (notshown) is optional, and where present would preferably be ahigh-precision spherical lens adapted to minimize distortion and otheraberrations for sharp and high-contrast images with a viewing anglerange of about 75°-150°. Preferred viewing angles are at least 90°, morepreferably at least 120°, still more preferably at least 150°, and mostpreferably at least 160°. Viewing angles 24A, 24B are shown as beingpointed directly forward and aft of surveillance vehicle 10, but canalternatively be pointed in other directions as well. The viewingdirection can optionally be motorized to scan a swath of area up, down,and sideways, or to point in a particular direction, and those functionscan be automated and/or manual. As drawn, forward camera 22A can “see”license plates 40A, and 42A, but not license plate 48A. Rearward camera22B can “see” license plate 44A, but not license plates 46A.

In FIG. 1 cameras 22A, 22B are ordinary video cameras. Other types ofcameras can be used, including still cameras, charge-coupled devicecameras for higher resolution, infrared cameras for night operations,and so forth. The focus is most likely set to infinite, but there canalso be an automatic focusing mechanism (not shown). One or both ofcameras 20, 22 can be advantageously provided with illumination, whichcan be in the form of a controlled light source (not shown) adapted tobrighten vehicle license plates during the day and/or allow cameraoperation during the night. Alternatively, the illumination means can bean infra-red (IR) light source, i.e. invisible to the driver of theneighboring vehicles.

In FIG. 2 surveillance system 20 generally comprises first and seconddigital video cameras 22A, 22B, a processor 24, memory 26, and an alertgenerator 28.

Processor 24 can be any suitable processor, including for example CPU(s)(central processing unit(s)) made by Intel Corp. (e.g., Pentium®,Xeon®), AMD (e.g., Athlon®), Motorola, IBM, etc., I/O (input/output)circuits, communication bus links, etc. Processor 24 receives digitalimage data input from cameras 22A, 22B, and processes the data softwareresident in memory 27. The software preferably includes an operatingsystem (OS) (e.g., Windows®, Linux®, Unix®, Free BSD®, etc.), andoptical character recognition programs.

Memory 27 can also advantageously include county-wide, state-wide,nation-wide, or even multi-country vehicle license plate number data, aswell as related information of interest such as law enforcement-relateddata. Information that is not available on-board the surveillancevehicle 10 can be accessed wirelessly from a remote facility 60. In thatcase system 20 would need to be adapted for wireless connection usingcommunication hardware 29.

Optical character recognition preferably occurs on board vehicle 10, butmay alternatively or additionally occur in the remote facility 60, orelsewhere. Any suitable OCR software can be used, such as that ofHi-Tech Solutions, currently available through www.htsol.com. Manysuitable OCR algorithms operate in three stages. The first stageinvolves vectorizing the captured (raster) image. The second stage dealswith isolation of the vectors that describe the raster image. The thirdstage performs the subsequent alphanumeric character recognition togenerate a plate string. More details on suitable theory, methods andalgorithms can be found on the World-Wide-Web at:http://www.cae.wisc.edu/˜woochull/course/lpr.html;http://www.cs.technion.ac.il/Labs/Isl/Project/Projects_done/cars_plates/finalreport.htm;http://www.singaporegateway.com/optasia/imps, or in numerous literaturesources such as, for example, “Computer Graphics: Principles andPractice”, Foley, van Dam, Flener, and Hughes, Addison Wesley, Reading,Mass., 1990; which are incorporated herein by reference.

To lower cost, the processor 24, memory 28, and communication hardware29 would generally be included as part of a laptop or other computer(shown generally as component 21) that had already been installed invehicle 10 for other purposes. Alternatively, processor 40, memory 42,GUI 50, local license plate number database module 44, and speaker 46can be implemented as an integral part of cameras 22A, 22B.

The network used to communicate with remote facility 60 could be thesame network already being used by computer 21, or could be some othernetwork. Transmission of license plate data between processor 24 andremote facility 60 can be encrypted using any suitable data encryptionalgorithms to ensure data security.

Processor 24 cooperates with one or more alert devices, such as speaker28A, computer display 28B, or vibratory interface 28C. Alternatively,processor 24 can utilize any other speaker (not shown) in the vehicle10.

The entire system 20 is preferably programmed to operate autonomously inbackground mode, i.e., without any input being required from theoperator. The operator is preferably only alerted where the system 20has identified a possible law enforcement-related problem using thecaptured license plate information. System 20 can be designed to operatecontinuously for an extended period of time while vehicle 10 ispatrolling the streets/highways, and can be turned on and off by theoperator as desired.

Multiple instances of system 20 can be installed and operated onmultiple surveillance vehicles for more efficient license plate numbercheck coverage, and the various systems may cooperate with each other byexchanging information. System 20 can also be operated in conjunctionwith global satellite or other positioning systems (not shown). Thus,for example, one squad car may identify a neighboring vehicle at a givenlocation, and another squad car may identify the same vehicle 30 minuteslater at another location. By calculating the distance traveled by thetargeted vehicle, the system could calculate the minimum speed that thetarget vehicle traveled during that time frame. The vehicle license datagathered by system 20 during routine surveillance patrols could also beused for other than law enforcement purposes, as needed and/orauthorized by local law regulations.

A contemplated method includes the following steps: The processor 26activates one or more cameras 22A, 22B, which capture images of nearbyvehicle license plates. The captured information is read by a framegrabber in each camera. Pixel output from the frame grabber(s) is passedby the processor 26 through image processing software algorithms toenhance the image, if necessary, detect the vehicle license plateposition, and through OCR software algorithms determine a license placenumber. The processor then checks the license plate number againstcorresponding data records stored in memory 28, or in remote licenseplate number facility 60, for potential law enforcement-related vehicleproblems. If a problem is identified, the operator (not shown) isalerted audibly, visually or in a vibratory manner using one or more ofspeaker 28A, computer display 28B, or vibratory interface 28C,respectively.

While the present invention has been described in detail with regards tothe preferred embodiments, it should be appreciated that variousmodifications and variations can be made in the present inventionwithout departing from the scope or spirit of the invention. In thisregard it is important to note that practicing the invention is notlimited to the applications described hereinabove. Many otherapplications and/or alterations can be utilized provided that such otherapplications and/or alterations do not depart from the intended purposeof the present invention. Also, features illustrated or described aspart of one embodiment can be used in another embodiment to provide yetanother embodiment such that the features are not limited to thespecific embodiments described above. Thus, it is intended that thepresent invention cover all such embodiments and variations as long assuch embodiments and variations come within the scope of the appendedclaims and their equivalents.

1. A method of surveilling multiple target vehicles from a surveillancevehicle, comprising: providing or operating the surveillance vehiclewith a camera system disposed to capture license plate information oneach of the target vehicles regardless of whether the target vehiclesare in a same lane as the surveillance vehicle, or in left or rightadjacent lanes to that of the surveillance vehicle, while thesurveillance vehicle and the target vehicles are all moving; andproviding or operating the surveillance vehicle with a computerprogrammed to use the license plate information to determine licenseplate numbers for each of the target vehicles, and alert an operator ofthe surveillance vehicle only upon discovering that there is a potentialproblem related to one of the moving vehicles, all without a need forinput from the operator.
 2. The method of claim 1, wherein a first oneof the target vehicles is traveling in the same lane as the surveillancevehicle, and concurrently a second one of the target vehicles istraveling in the left lane adjacent to that of the surveillance vehicle,and concurrently a third one of the target vehicles is traveling in theright lane adjacent to that of the surveillance vehicle.
 3. The methodof claim 2, wherein the computer is further programmed to discover thepotential problem at least in part by comparing the license platenumbers to data in a database.
 4. The method of claim 3, furthercomprising the computer accessing the database from a source local tothe surveillance vehicle.
 5. The method of claim 3, further comprisingthe computer accessing the database from a source distal to thesurveillance vehicle.
 6. The method of claim 2, wherein the camerasystem comprises a first camera mounted at a front portion of thesurveillance vehicle, and the second camera is mounted at a rear portionof the surveillance vehicle.
 7. The method of claim 2, furthercomprising alerting the operator of the surveillance vehicle of thepotential problem while the surveillance vehicle is traveling at a speedat least 50 kilometers per hour.
 8. The method of claim 2, furthercomprising, transmitting at least a portion of at least one of theimages to a remote facility for license plate recognition.
 9. The methodof claim 2, further comprising, transmitting at least one of the licenseplate numbers to a remote facility to determine the potential problem.10. The method of claim 2, further comprising using a sound to alert theoperator of the existence of the potential problem.
 11. The method ofclaim 2, further comprising using a non-auditory alert to apprise theoperator of the existence of the potential problem.